fix: merge predicate for concurrent writes #2291
Merged
+432
−31
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@Blajda :) |
|
I have found another bug: After the successful merge, the merge operation will return a DeltaTable, for this the new commit will be merged in the old snapshot, but the concurrent commit is not included in it, which results in a version mismatch error. I'll try to recreate this in an integration test. |
|
@JonasDev1 yup it's a known issue and there is a PR open for it in the write operation: #2280, but we probably need to address the commit advancement across the board |
|
I have added some merge concurrency tests where the version mismatch error also occurs |
ion-elgreco
force-pushed
the
merge-predicate-fix
branch
from
133bb69
to
227dcd3
Compare
Blajda
approved these changes
Mar 19, 2024
This was referenced Mar 23, 2024
Blajda
added a commit
that referenced
this pull request
Mar 23, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This merge request will change the commitInfo.operationParameters.predicate of the merge operation. This is required to allow conflict checking on concurrent commits. Before, the predicate was just the simple merge predicate like
source.eventId = target.eventId, but the conflict checker doesn't know these aliases and doesn't have access to the source df.So I now use the early_filter, which is also used before to filter a subset of the target table. Basically, the early_filter only contains static filters and partition filters that are converted to fixed values based on the source data. The early_filter can be None if no column/partition pre-filtering is possible, or if the merge contains a not_match_source operation. (See the generalize_filter function in the file).
The commitInfo predicate uses exactly this filter, except that the target alias is removed.
The predicate is used by the conflict checker, for example when there are multiple concurrent merges. If there is a predicate, the conflict checker will check whether the concurrent commit wrote or deleted data within that predicate. If the predicate is None, the conflict checker will treat the commit as a `read_whole_table' and interpret any concurrent updates as a conflict.
Example:
Target table with partition country
Merge with predicate
source.id = target.id AND target.country='DE'-> CommitInfo predicatecountry='DE'Merge with predicate
source.id = target.id AND target.country=source.country-> CommitInfo predicatecountry='DE' OR country='US'Merge with predicate
source.id = target.id-> CommitInfo predicate None (As full target table join is required)Related Issue(s)